Comparing Test-Taking Effort Between Paper-Based and Computer-Based Tests.

The article compares the trajectories of students' self-reported test-taking effort during a 120 minutes low-stakes large-scale assessment of English comprehension between a paper-and-pencil (PPA) and a computer-based assessment (CBA). Test-taking effort was measured four times during the test. Using a within-subject design, each of the N = 2,676 German ninth-grade students completed half of the test in PPA and half in CBA mode, where the sequence of modes was balanced between students. Overall, students' test-taking effort decreased considerably during the course of the test. On average, effort was lower in CBA than in PPA. While on average, effort was lower in CBA than in PPA, the decline did not vary between both modes during the test. That is, students' self-reported effort was higher if the items were easier (compared to students' abilities). The consequences of these results concerning the further development of CBA tests and large-scale assessments in general are discussed.

On generating plausible values for multilevel modelling with large-scale-assessment data.

Large-scale assessments (LSAs) routinely employ latent regressions to generate plausible values (PVs) for unbiased estimation of the relationship between examinees' background variables and performance. To handle the clustering effect common in LSA data, multilevel modelling is a popular choice. However, most LSAs use single-level conditioning methods, resulting in a mismatch between the imputation model and the multilevel analytic model. While some LSAs have implemented special techniques in single-level latent regressions to support random-intercept modelling, these techniques are not expected to support random-slope models. To address this gap, this study proposed two new single-level methods to support random-slope estimation. The existing and proposed methods were compared to the theoretically unbiased multilevel latent regression method in terms of their ability to support multilevel models. The findings indicate that the two existing single-level methods can support random-intercept-only models. The multilevel latent regression method provided mostly adequate estimates but was limited by computational burden and did not have the best performance across all conditions. One of our proposed single-level methods presented an efficient alternative to multilevel latent regression and was able to recover acceptable estimates for all parameters. We provide recommendations for situations where each method can be applied, with some caveats.

Cultural variation in the SES-gender interaction in student achievement.

Introduction: Is the socioeconomic gap in academic achievement larger among boys than girls? Several scholars have proposed such an interaction between socioeconomic status (SES) and gender. Prior empirical studies have yielded mixed evidence, but they have been conducted almost exclusively in Western countries. Here we propose the hypothesis that the SES-gender interaction is stronger in less gender-equal societies. Methods: We estimated the SES-gender interaction in 36 countries using data from two international large-scale assessments (PIRLS and TIMSS). The degree of gender equality was measured by the Global Gender Gap Index. Results: Consistent with the hypothesis, the SES-gender interaction was stronger in societies with less gender equality. Discussion: Our findings suggest that cultural factors determine how the socioeconomic achievement gap differs between boys and girls.

Investigating Planning and Non-Targeted Exploration in PIAAC 2012: Validating Their Measures Based on Process Data and Investigating Their Relationships with Problem-Solving Competency.

Problem-solving is a critical aspect of intelligence that has become increasingly important in modern society. Mapping out the determinants of success in problem-solving helps understand the underlying cognitive processes involved. This article focuses on two key cognitive processes in problem-solving: non-targeted exploration and planning. We generalize previously defined indicators of planning and non-targeted exploration across tasks in the 2012 Programme for the International Assessment of Adult Competencies and examine the internal construct validity of the indicators using confirmatory factor analysis. We also investigate the relationships between problem-solving competency, planning, and non-targeted exploration, along with the specific dependence between indicators from the same task. The results suggest that (a) the planning indicator across tasks provides evidence of internal construct validity; (b) the non-targeted exploration indicator provides weaker evidence of internal construct validity; (c) overall, non-targeted exploration is strongly related to problem-solving competency, whereas planning and problem-solving competencies are weakly negatively related; and (d) such relationships vary substantially across tasks, emphasizing the importance of accounting for the dependency of measures from the same task. Our findings deepen our understanding of problem-solving processes and can support the use of digital tools in educational practice and validate task design by comparing the task-specific relationships with the desired design.

Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks.

Automated scoring of free drawings or images as responses has yet to be used in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a TIMSS 2019 item. We are comparing classification accuracy of convolutional and feed-forward approaches. Our results show that convolutional neural networks (CNNs) outperform feed-forward neural networks in both loss and accuracy. The CNN models classified up to 97.53% of the image responses into the appropriate scoring category, which is comparable to, if not more accurate, than typical human raters. These findings were further strengthened by the observation that the most accurate CNN models correctly classified some image responses that had been incorrectly scored by the human raters. As an additional innovation, we outline a method to select human-rated responses for the training sample based on an application of the expected response function derived from item response theory. This paper argues that CNN-based automated scoring of image responses is a highly accurate procedure that could potentially replace the workload and cost of second human raters for international large-scale assessments (ILSAs), while improving the validity and comparability of scoring complex constructed-response items.

Investigating the measurement of academic resilience in Aotearoa New Zealand using international large-scale assessment data.

Academic resilience captures academic success despite adversity and thus is an important concept for promoting equity within education. However, our understanding of how and why rates of academic resilience differ between contexts is currently limited by variation in the ways that the construct has been operationalised in quantitative research. Similarly, comparing the strength of protective factors that promote academic resilience is hindered by differing approaches to the measurement of academic resilience. This methodological variation has complicated attempts to reconcile disparate findings about academic resilience. The current study applied six commonly used operationalisations of academic resilience that combined different thresholds of high risk and high achievement, to three international large-scale assessments, to explore how these different operationalisations impacted the findings produced. The context of Aotearoa New Zealand was chosen as a case study to further academic resilience research within this context and investigate how academic resilience manifests in an education system with relatively high levels of average achievement alongside low levels of educational equity. Within international large-scale assessment datasets, prevalence rates differed markedly across subject areas, grade levels, and collection cycles, as a function of the measure of academic resilience employed, while the strength of protective factors was more consistent. Thresholds that were norm-referenced produced more consistent findings across the different datasets compared to thresholds that were criterion-referenced. High levels of missing data prevented the analysis of some datasets, and differences in the way that key constructs were measured undermined the comparability of findings across international large-scale assessments. The findings emphasise the strengths and limitations of utilising international large-scale assessment data for the study of academic resilience, particularly within the Aotearoa New Zealand context. Furthermore, the study highlights that researchers' methodological decisions have important impacts on the conclusions drawn about academic resilience. Supplementary Information: The online version contains supplementary material available at 10.1007/s11092-022-09384-0.

Multistage Testing in Heterogeneous Populations: Some Design and Implementation Considerations.

A central challenge in international large-scale assessments is adequately measuring dozens of highly heterogeneous populations, many of which are low performers. To that end, multistage adaptive testing offers one possibility for better assessing across the achievement continuum. This study examines the way that several multistage test design and implementation choices can impact measurement performance in this setting. To attend to gaps in the knowledge base, we extended previous research to include multiple, linked panels, more appropriate estimates of achievement, and multiple populations of varied proficiency. Including achievement distributions from varied populations and associated item parameters, we design and execute a simulation study that mimics an established international assessment. We compare several routing schemes and varied module lengths in terms of item and person parameter recovery. Our findings suggest that, particularly for low performing populations, multistage testing offers precision advantages. Further, findings indicate that equal module lengths-desirable for controlling position effects-and classical routing methods, which lower the technological burden of implementing such a design, produce good results. Finally, probabilistic misrouting offers advantages over merit routing for controlling bias in item and person parameters. Overall, multistage testing shows promise for extending the scope of international assessments. We discuss the importance of our findings for operational work in the international assessment domain.

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data.

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students' performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.

COVID-19 Pandemic and Student Reading Achievement: Findings From a School Panel Study.

Since 2020, the COVID-19 pandemic had an impact on education worldwide. There is increased discussion of possible negative effects on students' learning outcomes and the need for targeted support. We examined fourth graders' reading achievement based on a school panel study, representative on the student level, with N = 111 elementary schools in Germany (total: N = 4,290 students, age: 9-10 years). The students were tested with the Progress in International Reading Literacy Study instruments in 2016 and 2021. The analysis focused on (1) total average differences in reading achievement between 2016 and 2021, (2) average differences controlling for student composition, and (3) changes in achievement gaps between student subgroups (i.e., immigration background, socio-cultural capital, and gender). The methodological approach met international standards for the analysis of large-scale assessments (i.e., multiple multi-level imputation, plausible values, and clustered mixed-effect regression). The results showed a substantial decline in mean reading achievement. The decline corresponds to one-third of a year of learning, even after controlling for changes in student composition. We found no statistically significant changes of achievement gaps between student subgroups, despite numerical tendencies toward a widening of achievement gaps between students with and without immigration background. It is likely that this sharp achievement decline was related to the COVID-19 pandemic. The findings are discussed in terms of further research needs, practical implications for educating current student cohorts, and educational policy decisions regarding actions in crises such as the COVID-19 pandemic.

Measuring reading behavior in large-scale assessments and surveys.

Large amounts of studies have shown that reading behavior is an important aspect for the development of students' reading skills. The construct reading behavior as examined in large-scale assessments and surveys within the field of empirical educational research is operationalized through a wide range of reading and reading-related aspects (e.g., reading time, reading frequency, print exposure, reading engagement, book genre preferences, knowledge of authors or book titles) and a broad array of measurement methods are used. The approaches to measure the same variable - namely reading behavior - differ fundamentally from each other, while at the same time, a clear concept that would help to classify the used measurement instruments and to interpret them in relation to the superordinate construct of reading behavior is missing. Therefore, the present article aims to give an overview of methods to measure reading behavior within the context of large-scale assessments and surveys, and to discuss how they were implemented. Finally, we make some suggestions on how it might be possible to relate the applied measurement approaches to each other, especially in terms of their content and theoretical relationship to the overarching construct of reading behavior.

Comparing group means with the total mean in random samples, surveys, and large-scale assessments: A tutorial and software illustration.

In many disciplines of the social sciences, comparisons between a group mean and the total mean is a common but also challenging task. As one solution to this statistical testing problem, we propose using linear regression with weighted effect coding. For random samples, this procedure is straightforward and easy to implement by means of standard statistical software. However, for complex or clustered samples with imputed or weighted data, which are common in survey analyses, there is a lack of easy-to-use software solutions. In this paper, we discuss scenarios that are commonly encountered in the social sciences such as heterogeneous variances, weighted samples, and clustered samples, and we describe how group means can be compared to the total mean in these situations. We introduce the R package eatRep, which is a front end that makes the presented methods easily accessible for researchers. Two empirical examples, one using survey data (MIDUS 1) and the other using large-scale assessment data (PISA 2015), are given for illustration. Annotated R code to run group to total mean comparisons is provided.

An Exploration of Factors Linked to Academic Performance in PISA 2018 Through Data Mining Techniques.

International large-scale assessments, such as PISA, provide structured and static data. However, due to its extensive databases, several researchers place it as a reference in Big Data in Education. With the goal of exploring which factors at country, school and student level have a higher relevance in predicting student performance, this paper proposes an Educational Data Mining approach to detect and analyze factors linked to academic performance. To this end, we conducted a secondary data analysis and built decision trees (C4.5 algorithm) to obtain a predictive model of school performance. Specifically, we selected as predictor variables a set of socioeconomic, process and outcome variables from PISA 2018 and other sources (World Bank, 2020). Since the unit of analysis were schools from all the countries included in PISA 2018 (n = 21,903), student and teacher predictor variables were imputed to the school database. Based on the available student performance scores in Reading, Math, and Science, we applied k-means clustering to obtain a categorized (three categories) target variable of global school performance. Results show the existence of two main branches in the decision tree, split according to the schools' mean socioeconomic status (SES). While performance in high-SES schools is influenced by educational factors such as metacognitive strategies or achievement motivation, performance in low-SES schools is affected in greater measure by country-level socioeconomic indicators such as GDP, and individual educational indicators are relegated to a secondary level. Since these evidences are in line and delve into previous research, this work concludes by analyzing its potential contribution to support the decision making processes regarding educational policies.

Estimation of item parameters and examinees' mastery probability in each domain of the Korean medical licensing examination using deterministic inputs, noisy and gate(DINA) model.

PURPOSE: Deterministic inputs, noisy and gate (DINA) model is one of the promising statistical means for providing useful diagnostic information about a student' level of achievement. Diagnostics information is core element for improving learning instead of selection. Educators often want to be provided with diagnostic information which how a given examinees did on each content strand, called diagnostic profiles. The purpose of this paper is to classify examinees in different content domains using the DINA model. METHODS: This paper analyzed data from the Korean medical licensing examination (KMLE) with 360 items and 3259 examinees. The application study estimate examinees parameters as well as item characteristics. The guessing and slipping parameters of each item were estimated. DINA model was conducted as a statistical analysis. RESULTS: The output table shows the examples of some items, which can be used for the check of item quality. In addition, the probabilities of being mastery at each content domain were estimated, which indicates the mastery profile of each examinee. Classifications accuracy for 8 contents ranged from .849 to .972 and classification consistency for 8 contents ranged from .839 to .994. As a result, classification reliability in a CDM was very high for 8 contents in KMLE. CONCLUSION: This mastery profile can be useful diagnostic information for each examinee in terms of the content domains of KMLE. The master profile from KMLE provides each examinee's mastery profile in terms of each content domain. The individual mastery profile allows educators and examinees to understand that which domain(s) should be improved for mastering all domains in KMLE. In addition, the results found that all items are reasonable level with respect to item parameters character.

When Large-Scale Assessments Meet Data Science: The Big-Fish-Little-Pond Effect in Fourth- and Eighth-Grade Mathematics Across Nations.

The programming language of R has useful data science tools that can automate analysis of large-scale educational assessment data such as those available from the United States Department of Education's National Center for Education Statistics (NCES). This study used three R packages: EdSurvey, MplusAutomation, and tidyverse to examine the big-fish-little-pond effect (BFLPE) in 56 countries in fourth grade and 46 countries in eighth grade for the subject of mathematics with data from the Trends in International Mathematics and Science Study (TIMSS) 2015. The BFLPE refers to the phenomenon that students in higher-achieving contexts tend to have lower self-concept than similarly able students in lower-achieving contexts due to social comparison. In this study, it is used as a substantive theory to illustrate the implementation of data science tools to carry out large-scale cross-national analysis. For each country and grade, two statistical models were applied for cross-level measurement invariance testing, and for testing the BFLPE, respectively. The first model was a multilevel confirmatory factor analysis for the measurement of mathematics self-concept using three items. The second model was multilevel latent variable modeling that decomposed the effect of achievement on self-concept into between and within components; the difference between them was the contextual effect of the BFLPE. The BFLPE was found in 51 of the 56 countries in fourth grade and 44 of the 46 countries in eighth grade. The study provides syntax and discusses problems encountered while using the tools for modeling and processing of modeling results.

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects.

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000-2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

A preliminary pan-European assessment of pollution loads from urban runoff.

Acknowledging the difficulty of modelling pollution conveyed by urban runoff, this contribution presents a first pan-European quantification of loads from this diffuse source. We estimate annual loads of 5-days biochemical oxygen demand (BOD5), nitrogen (N), phosphorus (P) and total suspended solids (TSS) using a simple event mean concentration (EMC) model initially proposed by Heaney et al., 1976. On a European scale, this yields discharges corresponding to untreated wastewater of about 31 million population equivalents (PE) for BOD5, about 18.5 million PE for N and P and about 280 million for TSS. These represent 51% of the pollution coming from treated wastewater for BOD5, 15% for N and P and 461% for TSS. Although the model applied for the assessment was developed more than 40 years ago, the results are consistent with those obtained using more recent parameterizations, except for a tendency to underestimate P loads. Although lack of data on pollution from urban runoff makes model verification impossible, and the uncertainty on EMC models is known to be very high, urban runoff emerges as a significant source of pollution, and should be properly addressed as such. Reducing runoff volume from urban areas through improved water retention is not only key to pollution control, but also a no-regret option thanks to its co-benefits, especially when incorporated at early stages of planning and design.

Identification of Factors Associated With School Effectiveness With Data Mining Techniques: Testing a New Approach.

The study of school effectiveness and the identification of factors associated with it are growing fields of research in the education sciences. Moreover, from the perspective of data mining, great progress has been made in the development of algorithms for the modeling and identification of non-trivial information from massive databases. This work, which falls within this context, proposes an innovative approach for the identification and characterization of educational and organizational factors associated with high school effectiveness. Under a perspective of basic research, our aim is to study the suitability of decision trees, techniques inherent to data mining, to establish predictive models for school effectiveness. Based on the available Spanish sample of the PISA 2015 assessment, an indicator of the school effectiveness was obtained from the application of multilevel models with predictor variables of a contextual nature. After selecting high- and low-effectiveness schools in this first phase, the second phase of the study was carried out and consisted of the application of decision trees to identify school, teacher, and student factors associated with high and low effectiveness. The C4.5 algorithm was calculated and, as a result, we obtained 120 different decision trees based on five determining factors (database used; stratification in the initial selection of schools; significance of the predictor variables of the models; use of items and/or scales; and use of the training or validated samples). The results show that the use of this kind of technique could be appropriate if mainly used with correctly pre-processed data that include the combined information available from all educational agents. This study represents a major breakthrough in the study of the factors associated with school effectiveness from a quantitative approach, since it proposes and provides a simple and appropriate procedure for modeling and establishing patterns. In doing so, it contributes to the development of knowledge in the field of school effectiveness that can help in educational decision-making.

The Onset of Rapid-Guessing Behavior Over the Course of Testing Time: A Matter of Motivation and Cognitive Resources.

Digital tests make it possible to identify student effort by means of response times, specifically, unrealistically fast responses that are defined as rapid-guessing behavior (RGB). In this study, we used latent class and growth curve models to examine (1) how student characteristics (i.e., gender, school type, general cognitive abilities, and working-memory capacity) are related to the onset point of RGB and its development over the course of a test session (i.e., item positions). Further, we examined (2) the extent to which repeated ratings of task enjoyment (i.e., intercept and slope parameters) are related to the onset and the development of RGB over the course of the test. For this purpose, we analyzed data from N = 401 students from fifth and sixth grades in Germany (n = 247 academic track; n = 154 non-academic track). All participants solved 36 science items under low-stakes conditions and rated their current task enjoyment after each science item, constituting a micro-longitudinal design that allowed students' motivational state to be tracked over the entire test session. In addition, they worked on tests that assessed their general cognitive abilities and working-memory capacity. The results show that students' gender was not significantly related to RGB but that students' school type (which is known to be closely related to academic abilities in the German school system), general cognitive abilities, and their working-memory capacity were significant predictors of an early RGB onset and a stronger RGB increase across testing time. Students' initial rating of task enjoyment was associated with RGB, but only a decline in students' task enjoyment was predictive of earlier RGB onset. Overall, non-academic-school attendance was the most powerful predictor of RGB, together with students' working-memory capacity. The present findings add to the concern that there is an unfortunate relation between students' test-effort investment and their academic and general cognitive abilities. This challenges basic assumptions about motivation-filtering procedures and may threaten a valid interpretation of results from large-scale testing programs that rely on school-type comparisons.

Avaliação de larga escala e da aprendizagem na escola: um diálogo necessário / Large-scale assessment classroom and school: a necessary dialogue / Evaluación en larga escala y del aprendizaje en la escuela: un debate necesario

O artigo apresenta e discute o debate teórico acerca das aproximações e afastamentos das avaliações de larga escala e da aprendizagem na escola. Parte-se do pressuposto que a avaliação voltada aos processos educativos deve considerar as diferentes perspectivas que cada uma dessas dimensões avaliativas envolve. O texto assinala a importância da avaliação de larga escala como possibilidade de diagnóstico educacional, de orientação de políticas públicas e indica também que a avaliação da aprendizagem, no contexto da sala de aula, pode apresentar análises mais profundas e mais consequentes para as práticas pedagógicas. No entanto, as autoras, indicando as diferentes lógicas que cada uma dessas avaliações envolve, procuram afirmar a importância de se considerar metodologias, procedimentos diferenciados em sala de aula e não uma repetição das avaliações de larga escala, como usualmente vem acontecendo em nossas escolas. Partindo de uma perspectiva histórica dos modelos de avaliação educacional o texto debate as dimensões avaliativas e discute a importância de a escola promover um diálogo entre elas e não reduzir a avaliação da aprendizagem, no contexto da sala de aula, ao uso dos procedimentos e instrumentos da avaliação de larga escala.

The article presents and discusses the theoretical debate about the approaches and distancing of the large-scale and assessment of learning. It's assumed that the assessment focused on the educational processes must consider the different perspectives that each one of these evaluative dimensions involves. The text highlights the importance of large-scale assessment as a possibility of educational diagnosis and orientation of public policies, and also indicates that the evaluation of learning in the context of the classroom can present deeper and more consistent analyzes for pedagogical practices. However, the authors, indicating the different logics that each one of these evaluations must involve affirm the importance to consider methodologies, differentiated procedures in classroom and not a repetition of large-scale assessment, like it usually happens in our schools. From a historical perspective of the educational assessment models, the text debates the evaluative dimensions and discusses the importance of the school to promote a dialogue among them and not to reduce the learning assessment to the usage of the large-scale assessment procedures and instruments.

El artículo presenta y discute el debate teórico acerca de las aproximaciones y alejamientos de las evaluaciones de larga escala, de la escuela y aprendizaje. Se supone que la evaluación que busca a los procesos educativos debe considerar a las diferentes perspectivas que cada una de estas dimensiones envuelve. El texto señala la importancia de la evaluación de larga escala como posibilidad de diagnóstico educacional y de orientación de políticas públicas, e indica también que la evaluación del aprendizaje, en el contexto del aula, puede presentar análisis más profundas y más consecuentes para las prácticas pedagógicas. No obstante, las autoras indicando las diferentes lógicas que cada una de estas evaluaciones envuelve, buscan afirmar la importancia de considerar a metodologías, procedimientos diferenciados en aula de clase y no una repetición de las evaluaciones de larga escala, como generalmente ocurre en nuestras escuelas. Partiendo de una perspectiva histórica de los modelos de evaluación educacional, el texto debate las dimensiones evaluativas y discute la importancia de la escuela promover un dialogo entre ellas y no reducir la evaluación de aprendizaje en el contexto del aula, al uso de los procedimientos e instrumentos de la evaluación a gran escala.

Mapping Background Variables With Sequential Patterns in Problem-Solving Environments: An Investigation of United States Adults' Employment Status in PIAAC.

Adult assessments have evolved to keep pace with the changing nature of adult literacy and learning demands. As the importance of information and communication technologies (ICT) continues to grow, measures of ICT literacy skills, digital reading, and problem-solving in technology-rich environments (PSTRE) are increasingly important topics for exploration through computer-based assessment (CBA). This study used process data collected in log files and survey data from the Programme for the International Assessment of Adult Competencies (PIAAC), with a focus on the United States sample, to (a) identify employment-related background variables that significantly related to PSTRE skills and problem-solving behaviors, and (b) extract robust sequences of actions by subgroups categorized by significant variables. We conducted this study in two phases. First, we used regression analyses to select background variables that significantly predict the general PSTRE, literacy, and numeracy skills, as well as the response time and correctness in the example item. Second, we identified typical action sequences by different subgroups using the chi-square feature selection model to explore these sequences and differentiate the subgroups. Based on the malleable factors associated with problem-solving skills, the goal of this study is to provide information for improving competences in adult education for targeted groups.